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The last decade has seen the emergence in the policy 
arena of a new form of qualitative research, one intended to 
strenghten its ability to generalize while preserving in-depth 
description. These multisite qualitative studies address the same 
research question using similar data collection and analysis 
procedures in a number of settings. To gain systematic knowledge 
about this phenomenon a study was undertaken of policy research 
projects in the field of education that were: (1) federally funded 
via a competitive request for proposals process, (2) involved the 
application of qualitative methods of data collection, and (3) 
compared several research sites. A two-step data collection strategy 
was employed. Telephone interviews were conducted with a key staff 
member of 25 projects. Subsequently documents were reviewed and site 
visits made to five projects. One important distinction among these 
multisite qualitative studies was the degree to which their 
methodology was "formalized." Formalization entails codification of 
the questions and variables to be studied, standardization of the 
data collection methods, and systematic reduction of verbal narrative 
to codes and categories. The variation in formalization observed 
across projects was substantial and seemed to result from different 
adaptations of academic social science to the policv research 
context. (Author) 
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The classical qualitative educational research design is 

the case study. Studies of school life (Cusick, 1973; Wolcott, 

PO 1973), of the larger social forces affecting schooling <Ogbu, 

O 

1974), and of efforts to promote planned educational change 
. , (Smith 8t Keith, 1971) have used qualitative data in describing a 

single social setting. Typically, such studies emphasize 
in-depth description but provide a weak basis for generalization 
to other settings. 

The last decade, however, has seen the emergence of a new 
form of qualitative research, one intended to strengthen its 
ability to generalize while preserving in-depth description. 
These njyitisite qualitative studies address the same research 
question in a number of settings using similar data collection 
and analysis procedures in each setting. They consciously seek 
to permit cross-site comparison without necessari ly sacrificing 
wi thin-site understanding. Although having some roots in 
academic social sciences (e.g., see Clark, 1970; Whiting, 1963; 
Whiting «c Whiting, 1975), multisite qualitative research arose 
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primarily in res'ponse to pressures from the federal government 
in the 1970s for studies that could overcome some of the 
weaknesses of large quantitative evaluations without being 
limited by the particularism of the single-site case study- 
Li many hybrids, it is today quite robust- However, these 
multisite qualitative studies were typically expensive endeavors 
and were done for specific policy purposes which the current 
federal administration seems neither to value no'^ to feel it can 
afford. 

There are two important reasons for reflecting on the 
historical development and potential utility of multisite 
qualitative policy research at this time. Although it is 
unwelcomed by most social scientists, the current hiatus in 
commissioning policy research at the federal level provides 
researchers and policy makers with an opportunity to consider 
these issues in some detail. Further, the field of policy 
research has matured to the point where such considerations can 
be very fruitful. In recent years qualitative researchers have 
moved beyond the need to defend the legitimacy of their craft in 
the policy arena (Rist, 1977; Smith, 1978; Stake, 1978). 
Moreover, quantitative researchers are beginning to acknowledge 
a role for qualitative research in policy and evaluation studies 
(Cronbach, 1982; Hoaglin, Light, McPeek, Mosteller & Stoto, 
1982) and to consider the proper balance of qualitative and 
quantitative techniques (Cook ic Reichardt, 1979; Smith it Louis, 
1982). In addition, practi tione»^s of multisite qualitative 
policy research now exhibit sufficient confidence in their 
craftsmanship to begin a process of public self-criticism with 



an eye to improving their methods (Firestone :Sc Herri ott, 1983; 
Miles, 1979; Smith & Louis, 1982; Yin, 1981). 

Efforts to examine multisite qualitative policy research 
suffer, however, from the absence of descriptive data about the 
field's status and growth. While there are useful first-person 
accounts of individual projects (e.g., see Fetterman, 1982; 
Herriott, 1982) the field lacks systeaiatic knowledge about a 
range of studies. The sections that follow offer a start in 
that direction. First we review the historical context 
associated with the use of this innovative design in the 1970s 
and present the results of a formal survey of 25 studies to 
highlight some of its institutional and methodological features. 
We then examine in detail the degree of "formalization" within 
five of these studies. Finally we consider ways in which 
current understandings of the strengths and weaknesses of 
multisite qualitive methods might be extended by academically 
oriented social scientists. 

HISTORICAL CONTEXT 
The tremendous growth of social programs in the 1960s led 
to even greater growth in the sponsorship of research about them 
in the 1970s. Initially, federally funded policy research 
concentrated on program outcomes, but gradually an expansion 
occurred to include an interest in program processes and 
implementation. Associated with this broadening of interest was 
a shift from the use of research designs that were exclusively 
quantitative to those that mixed quantitative and qualitative 
techniques and even to ones that were exclusively qualitative. 



In the sections which fallow we describe factors that 
contributed to this expanded interest in qualitative policy 
research and consider concerns about its validity and utility. 
Ihe iQterest in Ouaiitatiye Research 

Increased interest in qualitative research within the 
policy arena seems to have stemmed largely from a reaction 
against exclusively quantitative approaches. The reasons for 
this reaction have not been well documented , but involve a 
mixture of considerations focused on issues of political 
utility, scientific validity and forms clearance. One federal 
official notes a concern that early evaluation designs in the 
field of education were "findings poor" because they could not 
help policy makers understand why programs like Head Start and 
Follow Through had null effects or how to improve them (Datta^ 
1982). Another argues that the discontent was also with the 
content of the findings, "few of which were liked by program 
advocates" (Smith, personal communication, 1982). 

The validity concerns with quantitative studies related to 
both outcomes and "treatments." In education, for example, the 
outcome problem focused on the potential bias in existing 
measures of pupil performance which, it was argued, mitigated 
against showing positive effects (with even the strongest 
quantitative designs) for the minority group members who were 
the object of the most ambitious federally funded efforts 
(Cohen, 1975). In a variety of fields, there was also a 
question as to whether the treatments (i.e«, the federal 
programs) were sufficiently faithful to the intentions of their 
designers or enacted in a sufficiently uniform manner across 
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sites to permit a meaningful test of their effects (Weiss & 
Rein, 1970). This latter concern in particular contributed to a 
growing interest in the study of program implementation 
(Pressman & Wildavsky, 1973). 

A further contributor to the interest in qualitatiave 
studies was the growth of a cumbertr.ome forms clearance process 
for instruments used in federal evaluation contracts. 
Introduced originally to protect private industry from redundant 
federal data collection for regulatory purposes, forms clearance 
by the Office of Management and Budget in the 1970s was 
embroiled in issues of federal -state relations and of individual 
privacy. The review process became a major obstacle to 
standardized data collection efforts— of ten requiring delays of 
six months or more between initial study design efforts and the 
initiation of data collection (Carter, 1977; Datta, 1982). 

Over the course of the decade, qualitative studies 
increasingly came to be seen as a way to overcome the apparent 
limitations of quantitative studies. A former official of the 
National Institute of Education reports that qualitative studies 
were attractive to policy makers and program advocates because 
"case study approaches . . . tend to yield less controversial 
findings, ones with conclusions on both sides of a political 
decision" (Smith, personal communication, 1982). Qualitative 
studies also assisted in efforts to reconceptual ize the issue of 
program implementation (Greenwood, Mann & McLaughlin, 197fj) . 
One official responsible for the Follow Through planned 
variation experiments reports an increasing reliance on 
qualitative studies to understand why Follow Through "models" 
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were implemented so differently in different schools (McDaniels, 
personal communication, 1982). Finally, by the end of the 
decade, many federally funded projects, particularly in the 
field of education, relied at least in part on unstandardized 
data collections to minimize or eliminate the "forms clearance 
hassle- " 

Concerns about Qual i tative Research 

Inspite of such pressures favoring the increased use of 
qualitative studies in federally funded policy research, 
qualitative approaches in their most highly developed academic 
form — that of anthropological ethnography — were often seen as 
having limited applicability to the policy context. A 
congressional aide, who later became an official at the National 
Institute of Education (NIE) , reflected on the early experiences 
of that agency with the sponsorship of multisite ethnographic 
studies and openly questioned their relevance to the immediate 
needs of policy makers (see Mulhauser, 1975). However, an 
experience ethnographer, who was serving as an advisor to one of 
NIE's contractors, argued that ethnography would lose its 
credibility as a form of scholarship if it attempted to be 
evaluative (see Wolcott, 1975) « 

Concurrently, guantitatiye researchers questioned the ways 
that qualitative methods dealt with the problems of 
general izabi 1 ity and reliability (e.g., see Campbell, 1974). 
Their concern was with the larger domain, if any, to which the 
findings from qualitative policy research could be applied. 
Often the question focused on the relationship of the sample 
under study to a larger population of policy interest 



(generalization from sample to population), but it also focused 
on the relationship of what was being learned in individual 
sites to that at the "typical" location (generalization from 
case to sample). Moreover, some researchers and policy makers 
were quick to note that such a concern about "statistical 
general izability" failed to consider the fact that policy makers 
seldom were concerned solely about the effects of a specific 
treatment on a specific population at a specific point in time. 
Rather they were continually attempting to extrapolate from 
current experience to future aspirations. Cronbach (1982, p. 
76), for example, argues that "the evaluation of a program 
/should lead/ to a statement about what to expect if a certain 
plan of action is adopted (or continued) in a certain site or 
class of sites." Such forecasting requires inferences that go 
well beyond statistical generalization and is problematic for 
all forms of research. 

Traditional ethnography ignores issues of generalization 
and forecasting; it is 'radically particularistic. Spradley S« 
McCurdy (1972, p. 3) define ethnography as "the task of 

describing a particular culture" and differentiate it from 

■ft 

ethnology which compares and explains. To Wolcott (1975, p. 
112) "An ethnography is, literally, an anthropologist's 
'picture' of the way of life of some interacting group." Such a 
research tradition avoids efforts to explain, generalize, or 
draw 1 essons for appl i cati on i n other setti ngs. 

Sociologists doing qualitative research tend to be more 
willing to go beyond description. They have written a great 
deal about how qualitative research can be used to build theory, 
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including concepts and explanations. Glaser and Strauss (1967), 
for example, do not limit themselves to the study of a single 
"case". They and other qualitatively oriented sociologists 
proceed by generating explanations about a single social system 
intuitively and then disaggregating that case to individuals or 
events in order to seek confirmation or disconf irmation 
(Campbell, 1975). 

One problem with this approach is that it throws very 
little light on general i zabi 1 i ty beyond the particular case or 
on the conditions under which explanations derived from that 
case are likely to hold. All generalizations are, of course, 
tentative. However, one federal official suggested that the 
strenght of generalizations from case studies can be increased 
when many cases are included and the sample meets such criteria 
as substantial variety among cases, many similarities to the 
larger population of interest, and few unique characteristics 
(Kennedy, 1979). This line of reasoning was a major force 
behind the increasing federal interest in mylticase qualitative 
studi es. 

The reliability issues for qualitative research has to do 
with the accuracy and stability of measurement. Quantitative 
researchers typically give great attention to these by carefully 
designing and documenting procedures and instruments (Selltiz, 
Wrightsman & Cook, 1976). In traditional gualitative studies, 
there is less prespecif ication of data collection procedures in 
order to permit the researcher to interact with the setting and 
gain insights in the process. This is one reason for the 
observation that qualitative research often increases construct 
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validity at the e>;pense of reliability (McGrathj, 19B2) . Such a 
view does not to imply that qualitative researchers are not 
concerned with accuracy- Rather they seek to improve it through 
nonquanti tati ve means. These include extensive immersion in a 
setting, triangulation to check insights and hypotheses via 
multiple sources;, socialization to a relativistic viewpoint and 
especially the habit of introspection to check against personal 
bias (Wolcott, 1975). 

Mhile the ethnographer's approach to accuracy is well 
accepted within the community of qualitative researchers, it was 
not viewed positively by most quantitative researchers, at least 
not initially (see Campbell 8t Stanley, 1966). Moreover, it 
encountered two problems in the policy world not faced in the 
academic world. The first stemmed from the adversarial nature 
of some policy research- Findings that are unpopular or 
disadvantageous to an interest group are often attacked on 
methodological grounds. The researcher must be able to describe 
and defend data collection and analysis procedures. Further the 
data fviay have to withstand extensive methdological critique and 
secondary analysis as has happened -uth two of the Coleman 
studies (see Mosteller & Moynihan, 1972; Hal li nan ic Qlneck, 
19S2) . Historically, qualitative researchers have had great 
difficulty disseminating their procedures and data in sufficient 
detail to make their studies amenable to either replication or 
secondary analysi s. 

The second problem associated with the reliability issue 
was specific to multisite research. Given the many sites needed 
to increase general izability, the researchers seemed to lack the 
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flexibility of single-site designs. With more than one site^ 



comparability of data collection, reduction and analysis 
procedures across all s.'tes tended to be given priority over 
in-depth description at individual sites. In general this was 
done to ensure that whatever similarities and differences were 
noted among sites stem from inter — setting rather than 
inter — researcher variation (Pel to & Pel to, 1978). 

Such questions about the validity and utility of 
qualitative research created new demands on qualitative policy 
researchers in the 1970s. They led to the introduction of the 
multisite design as a way to cope with the problems of 
general izability and to the "formalization" of those designs as 
a way to cope with that of reliability. 



To learn more about the expansion of multisite qualitative 
methods in the 1970s we undertook a formal telephone survey of 
federal officials and qualitatively oriented researchers. 
Through a snowball sampling process we identified 25 projects 
which: (1) were federally funded via a competitive "request for 
proposals" (RFP) process, (2) involved the application of 
qualitative methods of data collection within at least a major 
part of the overall design, and (3) intended to compare two or 
more research sites. 2 Although the sampling process was clearly 
nonrandom it led to the selection of 25 projects of considerable 
diversity. 

One of the most noticeable features of these projects is 
that whereas single-site case studies arise almost exclusively 
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within academia all but four of these multisite projects were 
located within the type of private research firm which at that 
time was specializing in guantitative policy research (Table 1). 
With only five exceptions each of these projects contained two 
or more distinct substudies with varying degrees of qualitative 
or quantitative emphasis- Overwhelmingly their qualitative 
substudies were imbedded within multi method endeavors having 
quantitative components as well, thus providing opportunities 
not only for cross-site qualitative synthesis, but for the 
integration of qualitative and quantitative data (see Louis., 
1982a). The funding for these projects was rather extensive 
(typically over one million dollars) and their duration lengthly 
(typically at least three years). 3 

While the intent of IDyitislte gual_i.tatiye policy research 
is to optimize description and general izabi 1 ity, there is a 
persistent tension between these two objectives which permeates 
all research (Cook S< Campbell, 1979; McGrath, 1982). In 
multisite qualitative research this tension seems to revolve 
around four design issues. The most prominent of these issues is 
the degree to which the data collection effort should be 
"structured". Cross-site comparison and generalization require 
researchers at all sites to use shared definitions of concepts 
and common data collection procedures to ensure that cross-site 
similarities and differences are characteristics of the sites 
and not the result of measurement procedures or researcher bias 
(Pel to & Pel to, 1978). Yet such standardization encourages 
researchers to ignore the unique aspects of each site and to 
overlook processes and contexts that may make special 



contributions to the phenomena of interest. They also encourage 
the researchers to impose their definitions of the situation 
through premature conceptualization (Blumer, 1969). 

A high degree of structuring of data collection is obtained 
through the use of closed-end, prec ided questionnaires and 
interview schedules. Unstructured modes of data collection 
include unobtrusive observation and schedule-free interviewing. 
These are the primary forms of data collection for most 
traditional case studies. Our snowball sampling process 
excluded projects that relied primarily on highly structured 
data collection. Nevertheless, when we examined the data 
collection procedures employed by a major qualitative study 
within each of these 25 projects, we were surprised to find that 
only five relied primarily on unstructured data collection 
techniques. The other 20 employed primarily a variety of 
semi -structured procedures, including site-visit guides which 
specify the questions that must be answered but not the specific 
data sources to be used, open-end interview guides, and 
instructions for focused observation (Table 2). Such methods 
require that research issues be well thought out in advance 
rather than being derived "in the field." This heavy reliance 
on semi -structured procedures is clearly a major departure from 
the traditional single-site case study approach. It seems to 
represent an accomodation in the direction of quantitative 
methods, one made in order to facilitate cross-site comparison. 

A second design issue concerns the number of sites to be 
studied. To a point, general izabi 1 i ty is enhanced by the 
inclusion of many sites (Kennedy, 1979). However, for any given 



budget level, increasing the number of sites limits the 
resources that are available for describing and analyzing events 
at any one site or for cross-site comparison. Within this 
sample, the fewest sites studied was three and the most was 60 
with a median of 11. The 25 studies seem to cluster into three 
distinct groups: those with three thru six sites (7 instances), 
those with eight thru 22 tsites (13 instances), and those with 30 
thru 60 sites (5 instances). The five studies with over 30 
sites raise an interesting question: how does one synthesize the 
mass of qualitative data from so many locations when attempting 
to draw generalizations? One risk in attempting such a 
cross-site analysis is that the analyst will draw on the sites 
selectively, thus reducing data complexity but at the expense of 
representativeness. One alternative to such selectivity is to 
quantify the qualitative data through the use of rigorous coding 
schemes so that formal statistical models can be used in 
carrying out the cross-site analysis. Yet such quantification 
can undermine the descriptive value of qualitative research that 
the multisite design is intended to exploit. 

A third issue is the length of time to be spent at each 
site for purposes of data collection. Long-term immersion 
(generally of over one year) is the hallmark of classical 
ethnography (Wolcott, 1975) and is an important means of 
ensuring valid description (Dawson, 1982). However, increasing 
the amount of time at any one site limits the resources 
available for studying other sites and for cross-site comparison 
and generalization. On-site presence in this sample of 25 
studies fell into three broad categories: one or two short 
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visits to each site (10 instances), several intermittent visits 
(7 instances);^ and more continuous field work (S instances). 

Finally, the research team can emphasize site-specific 
reporting or cross-site, issue-specific reporting. 
Site-specific reporting is a literary device that enhances 
description but tends to mask similarities and differences 
across sites, thereby inhibiting generalization. Cross-site, 
issue-specific reporting facilitates generalization, but often 
at the expense of site-specific context. Although most of the 
25 studies surveyed used both site-specific and cross-site 
qualitative reporting formats, 12 emphasized the former and 13 
the latter. 

FORMALIZATION IN QUALITATIVE RESEARCH 
While our survey provided a rough outline of the features 
of multisite qualitative studies, it. raised a number of 
questions about how those features were combined and the extent 
to which they were complementary. To learn more, we conducted 
our own multisite qualitative study of five of these projects. 
This more intensive examination suggested one pervasive 
dimension underlying the methodological arrangements used in all 
of the projects: formalization. Formalization affects three 
aspects o^ the research process. In particular: 

o Whereas traditional qualitative research tends to 
emphasize the discovery of relevant questions and 
variables while in the field, these multisite studies 
tended to emphasize the cgdif icati_on of questions and 
variables before beginning fieldwork. 
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o Whereas traditional qualitative research tends to 
emphasize unstructured questioning and observation, 
these multisite studies tended to emphasize the 
standardi^zati^gn of data collection procedures through 
the use of semi structured interview and observation 
protocols. 

o Whereas traditional qualitative research tends to 
emphasize extended presentation of verbal narrative, 
these multisite studies tended to emphasize the 
systematic reduction of verbal narrative to codes and 
categories. 

While any one of these shifts alone would constitute simply 
a minor adaptation to the policy arena, the simultaneous 
occurence of all three produced a radical transformation in the 
way qualitative research is conducted. This transformation has 
been driven in part by the need to coordinate data collection in 
many sites and to ensure responsiveness to a client's need for 
cross-site conclusions. In addition, some advocates of such 
coordination argue that problem-driven research using 
standardized techniques for data collection and analysis 
increases the truth or accuracy of qualitative research by 
responding to standards of validity and reliability traditionally 
associated only with guantitatiye research (Huberman Sc Miles, 
1983). To them, what we have characterized as "formalization" 
represents a major improvement in the way that qualitative 
research i s conducted and apprai sed . 

The advantages and disadvantages of this dramatic shift in 
the conduct of qualitative research are currently being debated 
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by many of the principals (e.g., see Louies, 1982a; Miles, 1979? 
Rist, 1980; Wolcott, 1980; Yin, 1981;). Our research does not 
enter that debate directly. Rather it seeks to inform it by 
describing the degree to which the five policy research projects 
we examined formalized their research approaches. In the 
sections which follow we present the organizational context and 
structure of each project, describe variation in the degree of 
formalization across them, and consider the utility of highly 
f ormal ized designs. 

Eiy^ Myiti^site Projects 

We began the intensive phase of our research by arraying all 
25 projects in terms of two variables obtained via the telephone 
survey: the number of sites and the length of time spent in 
collecting data at each site (see Table 3). We then selected for 
detailed study one project from each of the five cells where 
either variable was relatively high. 4 Each of the five projects 
used research teams to carry out qualitative field work at 
multiple sitet5 with the intent of makinc^ cross-site 
generalizations. They differed substantially, however, in their 
methodological approach. The five projects are: 

o The Rural ExBeri mental Schools (RES) Study. Initiated 
in 1972, this complex multimethod project at Abt 
Associates Inc. explored the utility of comprehensive 
change efforts for reforming schools. In one of its 
five major substudies, ethnographic field work was 
conducted in 10 rural school districts over a 
three-year period by full-time "on-site researchers" 
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trained in the discipline of anthropology or sociology. 
The field work was coordinated by Stephen J. 
Fitzsimmons, Robert E. Herri ott and Michael B. KaneS. 
o The ExBerienced-Based Career Education (EBCE) study. 
This research by The Huron Institute was inaugurated in 
1976 to learn if EBCE "models" developed by four 
regional educational laboratories would be effective 
when exported to a wide variety of public school 
settings. Attention was also given to learning about 
program implementation as a social process. Over a 
three-year period three social scientists made several 
short visits to 45 schools. The amount of time spent 
at each school site varied from one to 22 person-days. 
Field work was conducted by Peter Cowden, John 
DeSanctis and Eleanor Farrar with David Cohen serving 
as senior advisor, 
o The Career Intern Program (CIP) Stud/. The CIP program 
originated at one site as a promising way to train 
.ninority youth to be employable workers or enter higher 
education. In 1778 it expanded to four geographically 
scattered sites. Through a multi method study the RMC 
Corporation investigated what happens when an attempt 
is made to replicate the prototype in new settings, 
what produces "successful" program outcomes and what 
those outcomes were. For purposes of an ethnographic 
substudy, approximately seven rounds of two-week visits 
were made to each site by a trained anthropologist. 
Key senior staff members included David Fetterman, 
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Kasten Tallmadge and Peter Treadway. 

The Parental iQvglyefnent (PI) Study. Begun in 1978, 
this large-scale project conducted by System 
Development Corporation described the form and extent 
of parental involvement within four federal educational 
programs. Data were collected at 57 sites over a 
four — month period by half-time, on-site field 
researchers. The formal academic training of these 
field workers varied from the pre— bachelors to 
post-doctoral level. All field work was coordinated by 
a staff of social scientists which included Ward 
Keesling, Ralph Melarango, Al Rabbins and Allen S^ith, 
each of whom played an active role in cross- site data 
analysis. 

The Dissemination Efforts SuQBortlng School ICDQrgvement 
(DESSI) Study. This complex multi method study was 
commissioned in 1978 to reconsider assumptions 
underlying federal dissemination strategies, to learn 
how school districts undertake planned change^, and to 
examine whether the federal government should promote 
fidelity to externally developed program models or 
local adaptations. Under the direction of David P. 
Crandall, The Network Inc. coordinated the work of a 
series of subcontractors, one of whom undertook case 
studies of 12 schools- Field work of approximately 
eight days per site was carried out over a three-month 
period by Jo Ann Goldberg, A. Michael Huberman, Matthew 
B. Miles and Beverly Taylor, with Huberman and Miles 
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subsequently conducting the cross-site analyses. 

Three of these five projects <RES, EBCE & CIP) were 
supported by the National Institute of Education and two (PI 8< 
DESSI) by the Office of Planning, Budget and Evaluation in the 
Office (later Department) of Education- All five were carried 
out by private corporations and were multimethod endeavors which 
included quantitative surveys in addition to the "case studies" 
we focused on. The projects ranged in duration from 33 months 
(PI) to eight years (RES) and in total budget level from *1 
million (CIP) to S5 million (RES). 

Var i ati on i^n Fgrfnaiization 

As noted above, what we are referring to as "formalized" 
qualitative research projects tend to have more codified research 
questions at the beginning, more standardized data collection 
procedures-, and more systematic means to reduce verbal data to 
categories for analysis. Table 4 summarizes variation among the 
five projects we studied in terms of each of these definitional 
elements. 6 

The classical qualitative research begins with only the most 
tentative research problem, and the first days in the field 
become an important time for fleshing out an understanding of the 
phenomena of interest (Geer, 1969). Formalized qualitative 
research begins with well specified conceptual models and uses 
early field work to refine the conceptualization and either to 
check the feasibility of questions or primarily to collect the 
necessary data. Within the five projects we studied RES embraced 
the traditional ethnographic field work model most fully. 
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delegating the task of designing case studies to the individual 
on-site researchers, each of whom was an experienced field 
woruer. Thus, there was never a central guiding 
conceptualization for its qualitative research. The EBCE team 
reported to us that in retrospect they could see the seeds of 
their major findings in their earliest proposal — perhaps 
reflecting ideas they had developed in doing other studies of 
implementation — but neither they nor the CIP team developed any 
formal fr^^^Clecl conceptualization to guide the research. PI and 
DESSI operated very differently. One staff member from the PI 
vleam devoted the first few months of the project to generating a 
model which elaborated five dimensions of parental involvement; 
he devoted less attention to specifying its causes and 
consequences. The DESSI team developed a comprehensive model of 
the major variables thought to affect educational change efforts 
and explicated 34 research questions. The RES, EBCE and CIP 
teams each used early field work to become grounded conceptually, 
with RES and EBCE making explicit reference to using the first 
year to develop their theory along the lines suggested by Blaser 
and Strauss (1967). The PI and DESSI teams moved more quickly to 
collecting the data called for by their conceptual models, 
although those models were modified somewhat over time (see Table 
4, Indicators 1 & 2) . 

As noted in the previous section, data collection techniques 
can vary on a continuum from unstructed, where researchers simply 
observe and ask questions, to highly structured, where 
closed-ended precoded instruments are used. The RES study never 
had a centrally imposed structure for its qualitative data 

20 

^1 



collection. However, over time some of the on-site researchers 
became progressively more structured in their approach, but only 
one developed formal inverview guides (see Firestone, 1980)- In 
contrast such progressive focusing was the rule on EBCE and CIP- 
At the end of the first round of site visits the EBCE team took 
time to reassess its research objectives and to write position 
papers. They used the insights gained from that collective 
process to guide later field work. Field work for CIP was done 
in seven rounds of site visits. What was learned in the first 
was checked 1 ater . PI and DESSI rel i ed pri mari ly on 
semi— structured guides. PI developed theirs before the field 
work from the a Bri^ori conceptualisations. DESSI finalized forms 
after the first, brief round of site visits. In both cases field 
work was geared to completing those guides, and there was 
frequent monitoring by senior researchers on both teams to assure 
that adequate data were collected to answer each question at each 
sites (Table 4, Indicators 3 & 4) . 

Data reduction is the task of condensing information about 
each site to manageable proportions, and it too can vary in its 
prespectif ication, with more standardized modes generally thought 
to facilitate cross-site analysis. RES essentially left this 
task to the discretion of the individual on— site researchers, and 
no formalized procedures were used in CIP since, a single field 
worker covered all four sites. EBCE experimented with a number 
of techniques, including creating a three-ring binder for each 
site in which field notes were cut up and organized by standard 
topics, and the use of wall charts to portray sites and topics in 
matrix form. In PI each field worker prepared a narrative 
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summary of data and observations far his or her site. These were 
followed by site-specific syntheses done by the central str-Ff 
following a standard outline and using the summaries and various 
interview forms as data. Before completing its field work, the 
DESSI team generated "interim" summaries of some sites and a case 
study outline with detailed data displays including dummy tables 
and tentative causal flow charts. These were subsequently 
completed for each case (Table 4, Indicator 5). 

Generally, traditional qualitative approaches show their 
rigor through extensive presentation of data close to its raw 
form while formalized qualitative approaches emphasize presenting 
primarily higher order data, one or more steps removed from the 
original field notes. RES reported its qualitative site data 
through book-length case studies (e.g., see Clinton, 1979; 
Firestone, 1980). EBCE presented illustrative quotes and 
vignettes in the cross-site analysis, but the reader cannot form 
an understanding of any specific site (see Farrar, DeSanctis i< 
Cowden, 1980). CIP used a similar approach but presented 
chapter-length case studies of each site (see Fetterman, 1981). 
PI presented some site-specific vignettes, but displayed most of 
its data in extensive narrative tables with variables as rows and 
sites as columns (e.g., see Smith & Nernberg, 1981). DESSI 
prepared case studies which are available to interested 
reviewers, but its public document features summary graphic 
displays for specific sites that were distilled from field notes 
during case study development (see Huberman & Miles, 1982) — Table 
4, Indicator 6. 

The credibility of cross-site qualitative analysis can often 
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be increased by the use of explicit preplanned procedures, 
including rules and displays -for coded data, and by 
intersub jecti ve checks requiring that there be consensus within 
the research team on the accuracy of coding and analysis 
(Firestone & Dawson, 1982). RES did not use standardized 
procedures for cross— site analysis, but its use of multiple 
independent synthesizers of the case study narratives (and in one 
instance the simultaneous presentation of five syntheses in a 
single report — see Herriott Z< Sross, 1979) enhanced the 
credibility of its approach. EBCE and CIP relied on a similar 
form of intuitive cross-site analysis, although only one 
synthesis was done within each project. The use of a team of 
three researchers on the EBCE study provided some checks and 
created the opportunity for each researcher to have to defend his 
or her conclusions. Teamwork was less evident in the CIP case, 
but the overall project director aggressively reviewed and 
challenged all reports. PI required that all conclusions be 
apparent in cross-site analysis tables, and that both table 
entries and the verbal patterns be defended in formal analysis 
committee meetings organized by conceptual element and by program 
studied (Smith & Robbins, 1982). Within PI both attacks and 
defenses of conclusions were extremely sprited. The DESSI senior 
researchers developed complex and thorough procedures for sorting 
sites and variables and for displaying the results (Huberman & 
Miles, 1983). They checked each other's work, but not with the 
same degree of open review required by the group context of PI 
(Table 4, Indicators 7 & 8) . 

In order to summarize the narrative picture of these five 

23 

24 



ERIC 



projects we read across the eight rows of Table 4 several times 
to get a sense of the range of variation on each indicator. Ws 
then read down each of the five columns to discover the modal 
tendency within each project. Although our original intent wat: 
simply to divide the five projects into two ordered categories 
(low formalization and high formalization) the data reflected 
three (low, moderate, and high). The RES study stayed close to 
the traditional ethnographic approach by delegating the data 
collection and case study writing to individual on-site 
researchers and by deemphasizing standardized cross-site 
analysis. It was at the low extreme. DESSI and PI, with their 
early conceptualization, extensive instrumentation and 
standardized data reduction, analysis and reporting techniques, 
were at the opposite extreme. CIP and EBCE were intermediate 
(Table 4)7. 

Ibe Uti 1 iti;: of Highli;: Formalized Designs 

What can be said on the basis of our research about the 
utility of highly formalized multisite qualitative studies? In 
some ways such an assessment is premature because this approach 
is still so new. We have seen useful research conducted at all 
three levels of formalization that we observed. Nevertheless, 
formalization, at least to a point, seems to have distinct 
advantages. The development of an initial conceptual framework 
and its operational izat ion through a series of open-ended 
instruments is extremely useful for ensuring comparability in 
data collection across sites and responsiveness to the original 
research issues identified by the client. 
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The more formal data reduction and analysis techniques also 
facilitate drawing conclusions. They provide a much more precise 
language through which members of a reseach team and reviewers 
from the sites studied can describe and debate conclusions about 
specific settings and then about cross-site patterns. This 
language forces the team to confront differences of perception so 
that conclusions can be "audited" (Lincoln ic Buba, 1982), and the 
agreement of a group of well'-inf ormed experts becomes a major 
claim for the credibility of findings. 

Whether these techniques constitute a major advance in the 
reliability and validity of qualitative research is more open to 
question. By themselves, they cannot constitute stronger "proof" 
for the uninformed reader. A great deal of judgment goes into 
the development of the type of ratings utilized by both the PI 
and DESSI teams — much more than goes into the numbers analyzed in 
survey or experimental studies. The reader must take it on faith 
that these judgments are correct. Typically, such judgments are 
less well justified in the final report of a highly formalized 
study than in that of research using a more traditional 
ethnographic approach where substantial excerpts from original 
field notes are shared with the reader. The authors of some 
formalized studies point out that case study materials are 
available for external audit (e.g, see Huberman 8c Miles, 1982), 
but these are generally difficult to use by individuals who did 
not do the original field work* 

In sum, techniques of formalization in multisite qualitative 
studies have advantages and disadvantages as means to bolster the 
credibility and utility of a research report. Their wider use 
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will depend in part on time and cost implications- Because they 
are fairly expensive to employ, we venture the preditions that 
they will become an important part of the "tool kit" of multisite 
qualitative researchers without becoming the si^ne gua ngn of good 
practice. The issue for those who commission and conduct 
qualitative policy research seems to be one of deciding how much 
formalization is appropriate under what combination of various 
scientific and political conditions (Cronbach, 1982; Firestone S/. 
Herriott, 1983) - 

DISCUSSION 

The introduction of multisite qualitative research to the 
policy world was part of the methodological eclecticism that 
characterized that field as it expanded rapidly in the 1970s. 
Although this design had its precursors in academic social 
scieK.a, it was largely an invention of federally-funded contract 
resarch. By the end of the 1970s, multisite qualitative studies 
were a fragile part of the policy scene. There was clearly 
"something in the air" which made this type of study useful to 
federal research sponsors, but there was great ambiguity on the 
part of both sponsors and researchers on matters of study design 
and implementation (Firestone tt Herriott, 1983). From a 
historical perspective, the formalization that took place in the 
1970s was an adaptation to the demands of the policy context. 
Just as quantitative researchers were seeking to enrich their 
understanding by incorporating qualitative elements into their 
work (e.g., see Cook ?< Reichardt, 1979) so qualitative 
researchers borrowed some techniques and invented others in order 
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to address canons of good work widely accepted in the 
quantitative world (Smith & Louis, 1982). 

The current hiatus in the commissioning of large-scale 
policy research provides academically oriented social scientists 
with unusual opportunities to explore in some detail the 
strengths and weaknesses of this design. In the process 
multisite qualitative research may have to be adapted back to the 
academic setting which, unlike the federal policy contextji 
generally requires that research be done at more modest cost but 
with longer time lines. 

Academic researchers can facilitate the development of 
multisite qualitative research by examining in greater detail 
than was possible in this study a broad range of methodological 
issues. For example, one current need is to understand better 
the consequences of different staffing patterns for data 
collection. This is an especially important issue in qualitative 
research where the investigator is often the crucial "instrument" 
(Sanday, 1979). It may be useful to conduct research which 
compares alternative data collection patterns. One such pattern 
is the use of a single investigator to carry out all field work 
in all sites (see Metz, 1978). Such an approach standardizes the 
data collection "instrument" across sites without sacrificing the 
potential for in-depth description, but it seems limited to 
situation involving no more than three or four sites. An 
alternative possibility is to provide greater data collection 
structure across multiple field workers, either through the use 
of field manuals (Campbell Sc Levine, 1973) or by having the 
different field workers prepare case study narratives for their 
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sites using a common format agreed to after conducting some field 
work (see Herriott i< Gross, 1979). Although there has been some 
effort to compare such approaches, it has not been as systematic 
or as extensive as it could be (Perlman, 1973). It would also be 
useful to know the conditions under which it is preferable to use 
"local" residents or professional researchers as field observers 
and about the advantages and disadvantages of doing cross-site 
comparison and generalization with field workers collaborating 
"in committee" or with "outside experts" who work only with the 
site-specific case study narratives. 

Another crucial issue is the consequences of different 
approaches to the standardized reduction of unstandardi zed data. 
Such reduction is a necessary first step to any analysis within 
or across sites (Goetz LeCompte, 1981). The potential of any 
study for useful, valid description and generalization depends on 
the analysts' ability to reduce data to a manageable form without 
distortion or loss of meaningful detail. Studies with a large 
number of sites, or where the principal investigator is not 
intimately familiar with all locations, are especially dependent 
on their approaches to data reduction. While we currently have 
some craft discussion of how data reduction was done in specific 
projects, we need to know more about the advantages and 
disadvantages of the quantification of qualitative data (see 
Louis, 1982b; Talmage & Rasher, 1981), and of verbal tabular and 
graphic data reduction devices (see Huberman & Miles, 1983; Smith 
ti Nerenberg, 1981). Other issues in need of attention are the 
timing of site visits in light of the phenomena under study, 
examination of processes and outcomes at different programatic 
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levels (studenty classroom, school, district, etc.) and 
alternative modes of presenting the results of research to 
pol i cymakers. 

Due to the pressure of time, major methodological issues of 
the type illustrated above can seldom be addressed systematically 
in the course of policy studies. Academically oriented 
methodological studies represent an opportunity to more fully 
explicate the logic of this developing research farm and to 
examine in detail its utility in both academic and policy 
contexts- 
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NOTES 



1. This paper is an expansion of one prepared with support from 
The National Institute of Education under contract No. 
400-80-0019 and subsequently published by the American 
Educational Research Association — see Herriott & Firestone 
(1983). It does not, however, necessarily reflect the view of 
either agency. We are particularly indebted to Fritz Mulhauser 
of the Institute's staff for his unfailing facilitation of our 
research, and to AERA for permission to use materials for which 
they hold the copyright. 

2. The snowball sampling process began with several highly 
visible qualitative researchers (Karen S- Louis, Matthew B. 
Miles, Ray C. Rist, Robert Yin) and federal officials (Edward 
Glassman, Frederick Mulhauser, Marshall Smith, James Vaneckp) . 
Through their recommendations — and the recommedations of persons 
suggested by them — a roster of approximately 100 candidate 
projects was created. Subsequent telephone calls to a person 
more knowledgeable about each project led to the elimination of 
approximately 75 projects, in most cases due to a failure to 
satisfy all three of the sampling criteria. For those projects 
meeting all criteria arrangements were made for a one-hour 
telephone interview, generally with the project's director- At 
the time of the interview the informant was queried about his/her 
project using a highly-structured "project profile" sheet as a 
guide. After the interview was finished a draft copy of the 
complete profile was sent to the informant and modifications 
requested if necessary- After the full set of 25 profiles had 
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been created and reviewed, they were used to code each project in 
terms of a series of summary categories. The key informants then 
reviewed that coding and Buggested whatever further modification 
of the profile sheets or summary tables seemed warranted. 

3. For a detailed description of each of the 25 projects, see 
Herriott ic Firestone (1982, Appendix A). 

4. No effort was made to achieve a random sample of projects 
within each of the five relevant cell** of Table 3. Instead we 
endeavored to select a sample representative of the field of 
qualitative policy research in the 1970s by emphasizing variation 
on the following seven factors: the funding agency, the 
contractor organization, the date of contract award, the size of 
the contract, the lenght of the funding period, the previous 
experience of key federal monitors, and the disciplinary 
background of key project staff. We also gave priority to 
projects that our informants Tin the snowball sampling process 
suggested were methodologically sophisticated. For comparable 
data on all 25 projects, see Herriott & Firestone (1982, Appendix 
A) . 

5. The on-site researchers included Allan F. Burns, Charles A. 
Clinton, A. Michael Colfer, Carol J. Pierce Colfer, Milliam L. 
Donnelly, Ronald P. Estes, Jr., Will m A. Firestone, Lawrence 
Henni gh , Stephen J . Langdon , Donal d f ^ Messerschmi dt , Mar i 1 yn C. 
Richen, Charles I. Stannard, and C. Thompson Wacaster. In 
addition to their case study reports these anthropologists and 
sociologists produced a lively literature on the stresses and 
strains of qualitative field work in the policy research 
setting — see Herriott (1980, Appendix D) for illustrative 
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citations. 

6. To learn about each project we reviewed such documents as 
requests for proposals, the proposals themselves, assorted 
planning documents, final reports, and published books and 
articles. In four cases we conducted extensive interviews with 
key project staff at their offices to learn about things not 
apparent in the documents- The interviews focused on the natural 
history of each project, the interests of project staff and 
relevant outsiders, and a series of methodological and 
administrative dilemmas that we anticipated would arise 
frequently in multisite qualitative policy research- We spent 
from six to twelve hours talking with several members of each 
project team. (This step was not taken with the RES study since 
we had been members of its staff, Herri ott as the project's 
director and Firestone as one of the on-site researchers.) For 
all five projects we later talked to at least one of the federal 
officials responsible for its monitoring to better understand the 
project's history and to obtain a client perspective. 

Elsewhere we have considered three competing explanations 
for such variation in formalization: the technical requirements 
of the research, the demands of the research sponsors, and the 
interests of the research teams and their professional networks 
(see Firestone & Herriott, 1983). 
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Table 1. Distribution of 25 Federally Funded Policy Research 
Projects on Five Context Variables. 



Q9QtS>Lt Variable NyiCQber gf 

Ecoiects 



A. The contractor organi sation: 

Diversified privat« firm 
Special ized pri va firm 
University 



B. The number of distinct substudies: 

None (i.e., a single unified project) 5 



Two 
Three 



5 
9 



Four or more ^ 



C. The methodological emphasis: 

Exclusively qualitative g 

Primarily qualitative 12 

Equal qualitative and quantitative emphasis 3 

Primarily quantitative 2 



D. The project's total budgets 

Less than *500,000 6 

*500,000 to »1 million 6 

*1 to 2 million 5 

More than *2 million g 



E. The project's duration: 

Less than 2 years 7 

2 to 3 years 5 

3 to 4 years ^ 
More than 4 years 7 
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Table 2. Distribution of 25 Qualitative Policy Research Studies 
on Four Desi gn Var i abl es. 



Resign Variabl^e MUiDber of 

Studies 

A« The predominant data collection approach: 

Primarily semi -structured 14 

Semi -structured with some unstructured 6 

Primarily unstructured 5 

B» The number of sites being studied: 

Three thru si>: 7 

Eight thru twenty— two 13 

Thirty thru sixty 5 

Cm The degree of on— site presence;: 

One or two short visits 10 

Several intermittent visits 7 

Many repeated visits or continuous presence 8 

D. Analytic emphasis of report narrative: 

Primarily site-specific 12 

Primarily cross-site with some site-specific 3 

Exclusively cross-site 10 
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Table 3. Distribution of 25 Qualitative Studies by Length of Time 

on Site and the Number of Sites. (The five studies selected 
for intensive study are identified parenthetically,) 



Length of 
Time on Site 



2-5 



NyiOber of Sites 
6-15 



16+ 



One or two 
short visits 

Several inter — 
mittent visits 

Many repeated visits 
or continuous presence 



3 (CIP) 



5 (DESSI) 



3 (RES) 



5 (EBCE) 



(PI) 



Note: For the identity of all 25 studies, see Herri ott ?< Firestone 
(1982, Appendix A). 
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Table V« Descriptive Characterization of Five Projects on Eight Indicators of Formalization 



Indlcntor of 


Project 


PorMlUfiKlon 


RRS 


KKCK 






PI 


Cdiiflcation 

I. PurMl • priori 
Concaptualitatiofi 


Hxia 


i»plicit baaad ort think- 
ing on i«pla«antation 


NiniMal* Moitly fro* 
propoaal 


Detailed explication of 
tha Mjor variablae 
thought to effect aduca— 
tional chenge ef forte 


Detailed explication of 
fiva diaanalona of par- 
en tel invo|.veMcnt 


furposs of Ear-* 
licat Fiald tfork 


To bacosa grouadtd in 
tha aita and ita Inrgar 


To axplora tha phanos* 
anon of axparianca 
lifs^ caracr aducation 
at aach aita 


To beco«e ecqueinted 
with tha key pereonnel 
of each eite 


To collect initiel <3ete 
on the verioua e priori 
verieblee and refine 
the conceptualliatlon 


To collect initial date 
on eech of the five 
diaanaionii 


Standerdliatlo* 

DoMlnant Early 
Data Collaction 
Foniat 


Onatructurad obaarva- 
tion and intarviavins 


Unatructurad obaarva- 
tion and intarviawing 


Hoetly unetructurad 
interviewing end ob- 
eenretion looaely 


InforMl interviewing 
end obaenretion* eo«e 
eeHi-etnictured. 


Highly etYuctured exten- 
eive '*analyaie peckete** 
beeed on conceptualiia- 

tion 


Data CollacKioa 


Unatructurad ohaanra- 
tion and intanriawing 


Sani-atirueturad ofaaar- 
vation and intarvictf- 
ing 


Hoetly wiitructured 
interviwing ft obeer- 
vetioii guided by eaerg- 
ing concaptualisaCion 


Seai-etructured inter- 
view guide beeed on 

conceptualliatlon 


Seai-etructured exten- 
eive **enalyaie peckete** 
beeed on conceptueliia- 
tion 


Muetion/AiwlYfU 

(. Sita-apaelfie 
Data Radttctloa 
Approach 


Laft to diacratioii of 
aach fialdw^rkar at 
aach aita 


Tranafar of fiald notaa 
^iito a notabock for 
aach aita ft Tating of 
aach aita on a aarlaa 
of eaargant variablaa 


SuMMriae by eite on 
prograMatic topica 


Syeteaatic coding oi 
field notee end prepara- 
tion of a etenderdiied 
eite eunaiery for eech 
eite 


Site euMUiriee by fiald 
workere end eyntheeie 
by centrel eteff 


B. Sita-apaciflc 
1 Data Praaantatioa 

1 Approach 


Bxtfmdad nAnrativa caaa 
atudiaa for aighc aitaa 
Ariaf aa»i-atructurad 
caaa atudiaa for .fiva 
aitc* 


Praquant yaa of illua- 
tratlva quotaa and 
vignatt*a for ynidan* 
tifiad aitaa 


Chepter length caaa 
etudiee in topicel 
report 


Stenderdiiai cherte for 
eech eite 

8c«i-etructured caaa 
etudiee for each aite 


Etenderdiied verbel 
teblee coapering eitee 
on verieblee 
Praquent uee of illue- 
tretive quo tee end 
vignafctee 


7. Croaa-aita^Data 
^ Analyaia Froca- 
durca 


Traditional litaratura 
raviava hy varloua 
non*fiald vorkara of 
draft caaa atudiaa 


tntuitiva analyaia by 
the three fiald vorkere 
ee e teea 


Intuitive enalyaie by 
the eingle field 
worker reeponeible for 
croae-eite enalyaie 


Dieplay and eyateaatic 
analyeie of dete ueing 
pictoriel techniquee 
by two of the four 
field workere 


VOcMacXOn ok ■nv&jVAV 

coaaitteea of non- 
f ieldworkara to eye- 
taaatlcally eort eicaa 
and verieblee 


IntaraubJactAva 
Chccka on rata 
Reduction and 
Annlyaia 


Hultipla indap«ndant 
aynthaaiiata 


Xnfoniul diacuaeion by 
the two field workere 
whan on aite tripe 
Collaboretive review 
of field notea ft draft 
enalyiee by the three 
field workere 


Informl diecuaeion 
of fecte of each caae 
by reeearch tea* 


Colleborative review i 
of field notee end 
dreft analyeee 


Periodic diecueeion 
between field workere 
end their eupenrieore 
Creetion of "enelyeie 
coaaittaee** of euper- 
vieore 


Overall Index 
of " ^"latlon 


Low 


Moderate 


Hoderate 


High 


High 
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